determine eigenvalues of a 3x3 matrix